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ABSTRACT 


Typically,  analysis  of  remote  sensing  data  is  limited  to 
one  sensor  at  a  time  which  usually  contains  data  from  the 
same  general  portion  of  the  electromagnetic  spectrum.  SAR 
and  visible  near  infrared  data  of  Monterey,  CA,  were 
analyzed  and  fused  with  the  goal  of  achieving  improved  land 
classification  results.  A  common  SAR  decomposition,  the 
Pauli  decomposition  was  performed  and  inspected.  The  SAR 
Pauli  decomposition  and  the  multispectral  reflectance  data 
were  fused  at  the  pixel  level,  then  analyzed  using 
multispectral  classification  techniques.  The  results  were 
compared  to  the  multispectral  classifications  using  the  SAR 
decomposition  results  for  a  basis  of  interpreting  the 
changes.  The  combined  dataset  resulted  in  little  to  no 
quantitative  improvement  in  land  cover  classification 
capability,  however  inspection  of  the  classification  maps 
indicated  an  improved  classification  ability  with  the 
combined  data.  The  most  noticeable  increases  in 
classification  accuracy  occurred  in  spatial  regions  where 
the  land  features  were  parallel  to  the  SAR  flight  line. 
This  dependence  on  orientation  makes  this  fusion  process 
more  ideal  for  datasets  with  more  consistent  features 
throughout  the  scene. 
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I. 


INTRODUCTION 


Land  cover  classification  applications  of  remote 
sensing  exist  in  many  fields,  including  but  not  limited  to: 
civil  planning,  agriculture,  forestry,  and  tactical 
military  operations.  Civil  engineers  faced  with  rapid 
urbanization  and  population  growth  need  to  obtain 
information  about  ground  cover  and  type  to  adequately 
handle  urban  sprawl  (Thunig  et  al .  ,  2011)  .  It  has  been 
shown  that  data  fusion  from  multiple  sensors  of  different 
spatial  resolutions  can  be  performed  to  increase  the 
dimensionality  of  the  vectors  being  classified.  This 
approach  has  resulted  in  demonstrations  of  improved 
classification  accuracy  (Kempeneers  et  al . ,  2011). 


Remote  sensing  applications  range  from  specific 
material  identification  using  hyperspectral  imaging 
(spectral/spatial  imaging  in  many  contiguous  spectral 
bands) ,  to  elevation  mapping,  to  terrain  classification 
mapping.  Sensors  come  in  many  varieties  with  a  wide  range 
of  spectral  bands,  ground  sample  distances,  revisit  times, 
and  many  other  features.  New  applications  for  existing 
technologies  are  not  uncommon  in  the  field,  as  the 
potential  for  remote  sensors  is  often  far  greater  than 
imagined  during  the  design  stages. 


Data  fusion  is  the  concept  of  combining  data  from 
multiple  sources,  (i.e.,  imaging  radar  and  electro  optical 
multispectral  sensors)  in  order  to  simultaneously  exploit 
their  individual  phenomenologies .  The  goal  of  this  approach 
is  to  increase  identification  or  detection  abilities  and 
the  confidence  levels  of  both.  It  is  expected  that 
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employing  both  systems  will  reveal  more  information  about  a 
scene  than  either  system  is  capable  of  independently. 
Finding  new  ways  to  synchronize  these  data  should  increase 
the  information  gain  provided  by  this  synthesis. 

Early  work  in  synthetic  aperture  radar  (SAR) /electro 
optical  multispectral  data  fusion  began  in  1980  with  the 
combination  of  airborne  SAR  data,  an  airborne  multispectral 
scanner,  and  Landsat  data  (Guindon  et  al . ,  1980)  .  The 
classification  results  did  not  improve  in  this  first 
attempt  at  data  fusion  over  a  single  airborne  Multispectral 
Scanner  (MSS) .  In  1982,  Seasat  L-band  and  airborne  X-band 


SAR 

data 

were 

fused  with  Landsat 

multispectral  scanner 

(MSS) 

data 

to 

classify  land  cover 

type 

(Wu,  1982)  .  This 

work 

did 

show 

improved  classification 

results  however. 

unlike  the  work  in  this  document,  unsupervised 
classification  techniques  were  used.  In  1990  visible  and 
near  infrared  (VNIR)  MSS  and  SAR  data  were  used  to  classify 
different  vegetation  species,  densities,  and  even  different 
ages  of  the  same  species  successfully  (Leckie,  1990).  Many 
different  combinations  of  subsets  of  all  the  data  were 
tested  and  it  was  found  that  many  different  unique 
combinations  were  the  most  successful  discriminators  of 
certain  classes.  In  2011,  airborne  SAR  and  airborne  optical 
images  were  combined  to  classify  land  cover  types  with  a 
combined  classification  accuracy  and  Kappa  coefficient  of 
94%  and  0.92  respectively  (Liu  et  al .  ,  2011)  .  Similar  to 
the  work  in  this  document,  maximum  likelihood 
classification  was  used.  In  2012  it  was  demonstrated  that 
the  fusion  of  SAR  and  optical  data  could  be  used  to 
calculate  soil  moisture  with  high  accuracy  (Prakash  and 

Singh,  2012)  .  The  optical  data  were  used  to  create  a  land 
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cover  mask  of  the  area  and  the  additional  SAR  information 
related  to  the  NDVI  to  calculate  soil  moisture. 

A.  OBJECTIVES 

The  primary  objective  of  this  work  was  to  achieve 
improved  land  cover  classification  accuracy  by  using  two 
datasets  of  the  same  geographic  area  from  different 
portions  of  the  electromagnetic  spectrum. 

In  this  research,  multispectral  data  from 
DigitalGlobe' s  WorldView-2  (WV-2)  satellite  were  used  to 
classify  land  cover  types  for  a  portion  of  Monterey,  CA. 
Data  acquired  by  the  Uninhabited  Aerial  Vehicle  Synthetic 
Aperture  Radar  (UAVSAR)  were  also  analyzed,  and  the  two 
datasets  were  fused  for  combined  analysis.  The  fused  VNIR 
and  Radar  information  were  used  to  classify  the  scene  and 
the  results  were  compared  to  the  classification  from  the 
WV-2  data  only.  Classification  results  were  expected  to 
improve,  however,  due  to  spectral  variability  within  the 
scene  and  large  pixel  sizes  in  the  SAR  data,  classification 
improvements  were  nil  to  marginal. 
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II.  BACKGROUND 


A.  RADIATIVE  TRANSFER  FUNDAMENTALS 
1 .  Electromagnetic  Theory 

Electromagnetic  (EM)  radiation  is  a  form  of  energy 
that  is  emitted  and  absorbed  by  charged  particles.  This 
radiation  is  composed  of  electric  and  magnetic  fields  which 
oscillate  in-phase  perpendicular  to  each  other  and  the 
direction  of  wave  propagation.  The  energy  is  characterized 
by  the  frequency  of  these  oscillations.  The  collection  of 
frequencies  is  known  as  the  electromagnetic  spectrum 
(Olsen,  2007)  .  Figure  1  shows  the  names  and  associated 
frequencies  and  wavelengths  of  important  portions  of  the 
electromagnetic  spectrum  (Kleeman,  1930;  Olsen,  2007). 


Electromagnetic  radiation  can  be  described  as  a  stream 
of  photons  (massless  particles  moving  at  the  speed  of  light 
in  a  wavelike  pattern) (Kleeman,  1930) .  The  photons  have 
energy  related  to  the  frequency  of  their  wave  pattern  as 
shown  in  equation  1.1, 


E  = 


he 

T 


Equation  1  . 1 


where  c=2.99*10A8  m/s  is  the  speed  of  light,  and  h=  6.626 
mA2kg/s  is  Planck's  constant  (Olsen,  2007). 
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Figure  1.  The  electromagnetic  spectrum  (From  Wikimedia 
Commons  file  "Image:  Electromagnetic-Spectrum. png, " 
retrieved  June  1,  2012) 


Remote  sensing  utilizes  detection  of  the  phenomena  of 
the  interactions  between  materials  and  electromagnetic 
energy  at  different  wavelengths  in  the  spectrum  without 
actual  contact  with  the  materials  (Elachi,  1987) .  Commonly 
used  portions  of  the  spectrum  for  imaging  purposes  include 
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but  are  not  limited  to:  visible  light  (.4-. 8  micrometers), 
near  infrared  (.8-2.5  micrometers),  and  microwave  (usually 
Radar)  (1cm  -  lm) .  For  electro  optical  systems,  photons  are 
collected  on  sensor  arrays  that  are  calibrated  to  certain 
wavelength  ranges.  The  measured  intensity  value  recorded  by 
a  sensor  represents  the  number  of  photons  collected  and  is 
recorded  as  a  digital  number.  Measurements  are  often  taken 
across  an  entire  scene  and  compiled  to  create  a  dataset 
that  is  displayable  as  an  image  (Lulla  and  Jensen,  1987) . 

Remote  sensing  detects  electromagnetic  radiation  that 
has  interacted  with  the  materials  being  observed  in  some 
way  (scattered,  transmitted,  emitted,  or  reflected) . 
Usually  this  is  done  by  measuring  the  amplitude  of  the 
energy  at  certain  wavelengths.  Beyond  amplitude,  EM  waves 
also  carry  polarization  information,  which  can  be  used  to 
provide  addition  information.  Spectral  information  gives 
information  about  the  materials  being  imaged,  whereas 
polarization  informs  us  of  the  surface  features,  shape, 
shading,  and  roughness  (Tyo,  J.  Scott  2006)  .  The  SAR  data 
used  in  this  work  contain  polarization  information. 

As  an  EM  wave  propagates  in  a  given  direction,  the 
electric  and  magnetic  fields  that  compose  the  wave  are 
oscillating  perpendicular  to  the  direction  of  propagation 
and  to  each  other.  Polarization  is  the  orientation  of  the 
oscillations  of  a  wave.  Because  the  electric  and  magnetic 
fields  are  instantaneously  perpendicular  to  each  other,  it 
is  sufficient  to  describe  only  one  of  these  two  components. 
As  a  matter  of  convention  we  often  describe  the  electric 
field.  We  further  decompose  the  electric  field  vector  into 
two  orthogonal  components  for  a  complete  description  of  its 
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state  in  localized  2D  space.  The  Cartesian  x  and  y  unit 
vectors  are  the  most  common  basis  used  in  this  description 
(Graves,  1956)  . 

A  polarization  state  is  the  2-dimensional  shape  that 
is  traced  out  by  the  electric  field  vector  in  a  plane 
perpendicular  to  the  direction  of  wave  propagation.  If  the 
two  orthogonal  components  of  the  electric  field  are  in 
phase  the  ratio  of  the  amplitude  of  the  each  component  is 
constant  and  the  polarization  state  is  linear.  If  the  two 
orthogonal  components  have  the  same  amplitude  and  are 
ninety  degrees  out  of  phase  then  the  polarization  state  is 
circular.  All  other  cases  result  in  elliptical 
polarization.  The  three  possible  polarization  states  are 
depicted  in  Figure  2  (Graves,  1956)  . 
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Figure  2.  The  three  possible  polarization  states:  linear 
polarization  (top) ,  circular  polarization  (middle) ,  and 
elliptical  polarization  (bottom)  (From  Chai,  2011). 


Polarization  states  can  be  described  using  a  number  of 

methods.  One  such  method  is  the  parameterization  of  the 

polarization  ellipse  (Ward  and  Smith,  1974) .  These 

parameters  are  the  orientation  angle  i|j(0^ij;-n)  and  the 
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ellipticity  angle  x (_n/4<x^n/4 )  .  Figure  3  shows  these 
parameters  and  their  relation  to  the  polarization  ellipse. 
The  orientation  angle  is  the  angle  between  the  X  axis  and 
the  major  axis  of  the  ellipse.  The  ellipticity  angle  is 
the  angle  between  the  major  axis  and  a  chord  connecting  an 
intersection  of  the  ellipse  with  the  minor  axis  and  the 
intersection  of  the  ellipse  with  the  major  axis. 


Figure  3.  Polarization  ellipse (From  MacDonald,  1999) 

B .  MULT I SPECTRAL  IMAGING 

VNIR  multispectral  imaging  from  a  space-born  platform 
measures  the  intensity  of  the  electromagnetic  radiation 
emitted  from  the  sun  and  reflected  from  the  surface  of  the 
earth  to  the  sensor.  The  wavelength  bands  vary  from  sensor 
to  sensor.  Landsat  1  was  the  first  commercial  multispectral 
sensor  in  space  (Gupta,  1991).  This  sensor  collected 
measurements  in  seven  spectral  bands  from  .475  to  1.1 
micrometers  usually  in  .1  micrometers  bandwidths .  Three  of 
the  bands  were  centered  over  red,  green,  and  blue  to  create 
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true  color  images  using  the  RGB  color  scheme  and  a  near 
infrared  (NIR)  band  was  usually  included.  This  band  scheme 
has  stood  the  test  of  time,  most  subsequent  space  borne 
systems  and  current  state-of-the-art  sensors  such  as 
DigitalGlobe' s  Worldview-2  (WV-2)  sensor  provide  similar 
capabilities,  albeit  with  smaller  pixels.  WV-2  data  were 
used  for  the  electro  optical  component  of  this  study  (see 
capabilities  description  below) . 

Remote  sensing  from  space  introduces  new  problems  in 
classification  of  imagery.  Image  pixels  are  relatively 
large  as  a  result  of  the  Rayleigh  criterion  and  the  range 
of  space-born  sensors  from  the  surface  of  the  Earth.  The 
limiting  size  of  pixels  is  further  reduced  by  atmospheric 
turbulence  (Fried,  1966)  .  These  large  pixels  result  in 
mixed  pixels  where  multiple  targets  are  in  the  same  pixel 
and  the  spectral  signature  in  the  image  is  a  combination  of 
the  two  individual  signatures  (Smedes  et  al . ,  1975) . 

Another  problem,  known  as  spectral  variability,  is  the 
result  of  variation  in  nature  of  what  we  would  consider  the 
same  target  type. 

C .  SAR  IMAGING 

SAR  imaging,  unlike  multispectral  imaging,  is  an 
active  imaging  mode.  This  means  that  the  microwave 
wavelength  energy  is  generated  by  the  sensor  (the 
transmitting  and  receiving  antenna  can  either  be  separate 
or  one  and  the  same) . 

A  synthetic  aperture  radar  image  indicates  the  average 

returned  power  from  the  targets  in  the  location  defined  by 

each  pixel.  This  returned  power  is  different  for  each  pixel 

as  determined  by  the  radar  cross-sections  of  the  targets 
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located  in  each  pixel.  Radar  cross-sections  are  dependent 
on  a  number  of  features  including:  target  material,  size  of 
the  target,  relative  size  of  the  target  with  respect  to  the 
wavelength  of  the  transmitted  wave,  incident  and  reflected 
angles,  and  polarization  of  the  transmitted  and  received 
energy  (Tomiyasu,  1978) .  I  moved  all  citations  inside 
sentence.  Scan  to  make  sure  I  did  not  miss  any.  In  contrast 
to  multispectral  imaging,  these  image  values  are  determined 
largely  by  the  geometry  and  size  of  the  scattering  target. 

The  interactions  between  target  and  the  energy  are 
typically  distinguished  by  the  dominant  scattering  method 
involved  (Tomiyasu,  1978;  Cloude,  1985) .  There  are  three 
main  methods:  smooth  surface,  single  bounce,  double  bounce, 
and  volumetric  scattering.  Smooth  surface  scattering  occurs 
when  the  radar  energy  is  incident  on  a  smooth  surface 
relative  to  the  wavelength  of  the  wave.  The  reflected  angle 
is  approximately  the  incident  angle  and  the  returned  energy 
is  near  zero.  Rough  surfaces  (relative  to  the  wavelength) 
scatter  in  all  directions.  Some  of  this  energy  is  returned 
to  the  sensor  and  is  known  as  single  bounce  scattering.  In 
general,  rougher  surfaces  have  higher  backscatter,  again 
depending  on  SAR  wavelength.  Double  bounces  occur  when  two 
smooth  surfaces,  one  flat  on  the  ground  and  the  other 
vertical,  combine  to  reflect  at  a  very  high  intensity. 
Examples  of  double  bounce  include  buildings  and  vertical 
vegetation  such  as  tree  trunks.  Volumetric  scattering 
occurs  when  the  energy  interacts  with  a  layer  of  randomly 
oriented  scatterers.  This  is  most  common  in  vegetated  areas 
such  as  forests. 
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Polarimetric  SAR  data  are  often  recorded  in  a  2x2 
scattering  matrix  indicating  the  intensity  of  the  reflected 
radar  energy  with  respect  to  a  polarization  basis  (Cloude 
and  Pottier,  1996) .  The  scattering  matrix  can  be 
represented  using  an  infinite  number  of  basis 
representations.  A  common  basis  is  horizontal  and  vertical 
linear  polarization  which  results  in  a  scattering  matrix 
similar  to  the  one  in  equation  1.2 
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Shh  Shv 

S  ,  S 

vh  vv 


Equation  1 . 2 


where  the  first  and  second  subscripts  represent  the 
transmitted  and  received  polarization  respectively. 
(Huynen,  1990)  In  general  the  cross-polarization  elements 
shv  and  svh  are  equal  (Huynen,  19  90)  . 

These  data  can  be  difficult  to  interpret  and  a  number 
of  methods  called  decompositions  have  been  developed  over 
the  years  to  accomplish  this  task.  In  general,  targets  are 
distinguished  by  the  dominant  scattering  methods  discussed 
above.  SAR  decomposition  methods  aim  to  separate  these 
scattering  methods  in  order  to  distinguish  targets  in  the 
image  based  on  the  level  of  scattering  (Cloude  and  Pottier, 
1996).  See  the  Classification  section  for  additional 
details . 


D.  WORLDVIEW-2  SATELLITE  IMAGERY 

WorldView-2  (WV-2)  is  a  commercial  multispectral 
satellite  owned  by  DigitalGlobe .  The  satellite  is  capable 
of  .5  m  panchromatic  and  2  m  multispectral  spatial 
resolution  imagery.  The  WV-2  multispectral  data  are 
contained  in  8  spectral  bands.  In  addition  to  the  more 
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common  blue  (.450-. 510  micrometers),  green  (.510-. 580 
micrometers),  red  (.630-. 690  micrometers),  and  near- 
infrared  (IR)  (.770-. 895  micrometers),  WV-2  has  coastal 
(.400-. 450  micrometers),  yellow  (.585-. 625  micrometers), 
red  edge  (.705-. 745  micrometers),  and  NIR  2  (.860-1.040 
micrometers)  bands.  These  additional  bands  provide 
information  not  normally  collected  by  multispectral  sensors 
(Marchisio,  2010). 

The  multispectral  dataset  used  in  this  work  was 
collected  on  April  11,  2012,  using  the  DigitalGlobe  WV-2 
imaging  satellite.  The  image  scene  is  of  the  Monterey 
Peninsula  in  California.  The  image  size  is  8194  samples  by 
7135  lines  and  8  bands  at  2.25  meter  spatial  resolution 
shown  in  Figure  4.  Only  a  small  subset  of  these  data  (3120 
samples  and  2363  lines,  approximately  7x5  km)  were  used  for 
the  combined  analysis. 
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Figure  4.  WV-2  Dataset.  Red  box  outlines  subset  used  for 

this  study. 

E .  UNINHABITED  AERIAL  VEHICLE  SYNTHETIC  APERTURE  RADAR 
(UAVSAR) 

UAVSAR  is  a  polarimetric  L-band  (23cm)  airborne  SAR 
sensor  that  is  designed  for  repeat  track  collection  for  the 


purposes  of  interferometric  measurements 

(Chapman,  2010)  . 

The  system  is 

capable  of  2m 

range  resolution 

using 

the 

optimal  range 

bandwidth  of 

80  MHz. 

The 

antenna 

is 

electronically 

steerable  which 

allows  for 

more 

control 

of 

the  antenna  beam  independent  of  air  speed  (Chapman,  2010) . 
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The  SAR  data  set  used  for  this  work  was  collected  on 
November  14,  2009,  using  the  UAVSAR  L-band  sensor.  The 

scene  collected  includes  the  Monterey  Peninsula  along  with 
a  large  swath  of  California  to  the  northeast  along  the 
collection  flight  line  (flight  line  ID  23025)  .  The  data 
used  in  this  work  have  7.2  meter  azimuth  resolution  by  5  m 
range  resolution.  An  image  is  shown  in  Figure  5.  The  data 
are  distributed  via  the  UAVSAR  NASA  JPL  website 
(http : // uavsar . jpl . nasa . gov)  in  compressed  Stokes  matrix 
format . 


Figure  5 .  UAVSAR  dataset 
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F.  IMAGE  CLASSIFICATION 

One  way  in  which  information  is  extracted  from  image 
data  is  known  as  classification.  Image  classification  is 
usually  performed  on  a  per  pixel  basis  and  is  the  process 
of  assigning  each  pixel  into  a  class.  (Richards,  2005a) 
General  land  classes  used  for  classification  include  urban, 
vegetation,  and  water  among  others. 

Classification  methods  fall  into  one  of  two 
categories:  supervised  or  unsupervised  (Mohd  et  al .  ,  2009). 

Unsupervised  classification  procedures  separate  the  targets 
into  classes  without  any  a  priori  knowledge.  An  example  of 
such  a  method  is  k-means  (Hartigan  and  Wong,  1979)  . 
Conversely,  classification  can  be  supervised.  These  methods 
classify  the  image  into  predefined  classes,  usually 
determined  using  knowledge  of  the  ground  truth  (Richards, 
2005b) .  Examples  of  such  a  method  include  minimum  distance, 
maximum  likelihood,  and  Mahalanobis  distance  (Wacker  and 
Landgrebe,  1971;  Strahler,  1980;  McLachlan,  1999).  All 
analysis  in  this  document  was  accomplished  using  supervised 
classification  methods. 

1.  Supervised  Classification  Methods 

Supervised  classification  methods  are  the  most  common 
classification  procedures  used  in  remote  sensing 
applications.  All  supervised  classification  methods  include 
the  following  basic  steps: 

1.  Choose  the  set  of  classes  into  which  the  dataset 
will  be  segmented. 

2.  Select  training  data  for  each  class. 
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3.  Estimate  the  class  parameters  necessary  for  the 
chosen  classification  method  using  the  training  data. 

4.  Classify  each  pixel  in  the  image  using  the  selected 
classification  algorithm  (Richards,  2005b) . 

The  minimum  distance  classifier  has  the  benefit  of 
requiring  less  training  data  than  many  other  classifiers  in 
order  to  achieve  good  results  (Wacker  and  Landgrebe,  1971)  . 
The  discriminant  function  for  the  minimum  distance 
classifier  is  derived  from  the  squared  distance  formula  and 
has  a  final  form  of 


if  g;(x)>gy(x)  for  all  j*i  Equation  2.1 

where 


gi(x)  =  2mi-x-mi-mi ,  x  is  the  pixel  vector  being 
classified,  and  is  the  ith  class  (Wacker  and  Landgrebe, 
1972;  Richards,  2005b). 

The  maximum  likelihood  classifier  uses  the  variances 
of  the  target  classes  as  well  as  the  mean.  The  maximum 
likelihood  decision  rule  classifies  according  to 


xe®,  if  p(ax  |  x)  >  p(0)j  |  x)  for  all  j*i  Equation  2.2 

An  application  of  Bayes'  theorem  and  a  few  simplification 
steps  results  in  the  following  discriminant  function 


St  0)  =  -  In  1 2  ■ 1 1  -(x  -  mi )%'  (x  -  m, ) 


Equation  2 . 3 


where  mi  and  E(.  are  the  mean  vector  and  covariance  matrix  of 
the  data  in  class  coi  (Strahler,  1980;  Richards,  2005b). 

Another  common  supervised  classifier  is  the 

Mahalanobis  distance  classifier  (McLachlan,  1999). 
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This 


method  is  fast  while  maintaining  directional  sensitivity 
similar  to  the  maximum  likelihood  classifier.  The 
discriminant  function  for  the  Mahalanobis  distance 
classifier  reduces  to  (McLachlan,  1999;  Richards,  2005b) 

gi(x)  =  {x-mi),'L~l(x-mi)  .  Equation  2.4 

All  three  of  the  classifiers  discussed  above  are  used 
and  compared  in  the  work  contained  in  this  document.  A 
common  method  for  validation  called  the  confusion  matrix  is 
the  classification  validation  method  used  in  this  work.  A 
confusion  matrix  displays  information  about  the  actual  and 
estimated  classifications  (Congalton,  1991;  Amrinder,  2012; 
Richards,  2005b)  (Table  1)  .  The  numbers  indicate  rows  and 
columns  corresponding  to  the  "c"  classes  represented  in 
maps  A  and  B.  ptj  is  the  proportion  of  pixels  from  class  i 
in  map  A  classified  as  class  j  in  map  B.  Variables  pt  and 
pt  are  the  ratios  of  pixels  for  class  i  to  the  total  number 
of  number  pixels  for  class  i  in  maps  A  and  B,  respectively. 


Map  A 
categories 

Map  B  categories 

1  2 

C 

Total 

1 

Pu 

P 12 

•  •  • 

Pic 

Pi. 

2 

P  21 

P22 

.  .  . 

Pic 

Pi. 

c 

Pci 

P  c2 

Pcc 

Pc. 

Total 

Pa 

P.2 

,  .  - 

P.C 

1 

Table  1.  Sample  confusion  matrix  (From  Monserud,  1992) 

In  the  example  confusion  matrix  below  (Table  2),  "a" 

represents  the  percent  of  pixels  correctly  classified  from 
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class  1.  The  value  represented  by  "b"  is  the  percent  of 
pixels  belonging  to  class  1  that  were  incorrectly 
classified  as  class  2.  The  matrix  element  "c"  corresponds 
to  the  percent  of  pixels  belonging  to  class  2  that  were 
incorrectly  classified  as  class  1.  The  final  element  "d"  is 
the  percent  of  correctly  classified  pixels  from  class  2. 


Predicted 

Class  1 

Class  2 

Actual 

Class  1 

a 

b 

Class  2 

c 

d 

Table  2.  Example  confusion  matrix 


Additional  statistics  are  often  calculated  from  the 
confusion  matrix  to  summarize  the  information  contained  in 
larger  confusion  matrices.  These  include  measures  such  as 
the  overall  accuracy,  kappa  coefficient,  user's  accuracy, 
and  producer's  accuracy  (Congalton,  1991). 

Overall  accuracy  is  the  ratio  of  correctly  predicted 
pixels  to  the  total  number  of  pixels  (Amrinder,  2012; 
Richards,  2005b) .  Mathematically  this  is  computed  by 
calculating  the  weighted  average  of  the  diagonal  elements 
(Congalton,  1991). 

User' s  accuracy  is  a  measure  of  the  commission  error 
and  is  computed  on  a  per  class  basis.  The  total  number  of 
correctly  identified  pixels  in  a  class  is  divided  by  the 
total  number  of  pixels  classified  as  that  class.  This  ratio 
represents  the  probability  that  a  pixel  classified  as  that 
class  in  the  classification  image  correctly  represents  that 
pixel  on  the  ground  (Congalton,  1991) . 
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Producer' s  accuracy  is  a  measure  of  omission  error  and 
is  also  computed  on  a  per  class  basis.  The  total  number  of 
correctly  identified  pixels  for  a  class  is  divided  by  the 
number  of  pixels  in  that  class  in  the  reference  data.  This 
ratio  represents  the  probability  that  a  pixel  from  the 
reference  data  in  that  class  will  be  correctly  identified 
(Congalton,  1991). 

The  Kappa  coefficient  is  a  multivariate  statistics 
calculation  which  indicates  the  proportion  of 
correspondences  between  maps  after  chance  agreement  is 
removed  from  consideration  (Cohen,  1960) .  The  overall 
accuracy  of  the  confusion  matrix  shown  in  Table  1  is 

Po  =  I  pu  .  Equation  2 . 5 

i=i 

The  overall  proportion  of  chance  agreement  is 

c 

Pe  =X Pi.P.i  •  Equation  2.6 

/= 1 

The  Kappa  statistic  is  a  normalized  difference  between 
these  two  values  defined  as 

Po~  Pe  T-,  4_  ■  O  -7 

K  = - .  Equation  2.7 

1  ~Pe 

The  Kappa  statistic  is  useful  because  it  will  be  equal 
to  one  for  two  maps  with  perfect  agreement  and  close  to 
zero  when  the  maps  agree  approximately  as  much  as  would  be 
expected  due  to  chance  (Monserud,  1992). 

Each  of  these  summary  statistics  contains  different 
information  derived  from  the  confusion  matrix.  These 
statistics  can  also  indicate  different  levels  of  accuracy 
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from  the  same  confusion  matrix  (Congalton,  1991)  .  Table  3 
indicates  the  different  range  of  Kappa  coefficients  and  the 
degree  of  agreement  that  they  represent  as  defined  by 
Monserud  (1992) .  An  understanding  of  how  these  various 
summary  statistics  are  computed  and  their  meaning  is  needed 
in  order  to  determine  success  on  a  per  application  basis. 


Lower 

bound 

Degree  of 
agreement 

Upper 

bound 

<0.05 

No 

0.05 

0.05 

Very  poor 

0.20 

0.20 

Poor 

0.40 

0.40 

Fair 

0.55 

0.55 

Good 

0.70 

0.70 

Very  good 

0.85 

0.85 

Excellent 

0.99 

0.99 

Perfect 

1.00 

Table  3.  Kappa  coefficients  and  their  degree  of 

agreement  (From  Monserud,  1992) 

2 .  Pauli  Decomposition 

While  not  strictly  a  classification  per-se,  the  Pauli 
decomposition,  applied  to  SAR  data  is  a  method  for  breaking 
the  data  down  into  components  explaining  surface  scattering 
properties.  The  results  can  then  be  further  classified 
using  a  variety  of  classification  approaches.  The  Pauli 
decomposition  is  the  most  common  SAR  decomposition  (Zhang 
et  al . ,  2008).  This  decomposition  represents  the  scattering 

matrix  (Huynen,  1965)  as  three  components  representing 
single-bounce,  double-bounce,  and  volumetric  scattering 
mechanisms.  In  comparison  to  other  coherent  decomposition 
methods,  the  Pauli  decomposition  is  excellent  for  exposing 
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natural  targets,  but  not  ideal  for  highlighting  man-made 
targets  (Zhang  et  al .  ,  2008). 

The  scattering  matrix  [S]  can  be  written  as: 


s,.  s. 

"1  0" 

"1  0" 

"o  r 

[s]  = 

nh  hv 

S  r  S 

[_  vh  vv  _ 

=  a 

0  1 

+  P 

0  -1 

+r 

1  0 

Equation  2 . 8 

where  a  =  (sllh+sw)/>f2,p  =  (skh-sw)/ J2,  and  y  =  ^2sh 


3  hv 


The  Pauli  decomposition' s  dimensionality  of  three 
makes  it  simple  to  represent  visually  using  the  RGB  color 
scheme.  Often  the  a,  f3 ,  and  y  components  are  represented 
as  blue,  red,  and  green  respectively  resulting  in  an  image 
similar  to  the  example  shown  in  Figure  6. 


|S/)h+  Syvl  |S/)/T  Sw| 


Figure  6.  Example  Pauli  decomposition  image  of  the  San 
Francisco,  California,  area  (From  Lee  &  Pottier,  2009) 
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This  image  of  San  Francisco,  CA,  shows  the  water  as 
having  relatively  high  single  bounce  returns,  the  urban 
areas  having  mixed  returns  but  significantly  higher  double 
bounce  than  other  areas,  and  the  vegetated  areas  as  having 
a  high  volumetric  scattering  return. 

3 .  Data  Fusion 

Data  fusion  is  the  combination  of  data  from  multiple 
sources  motivated  by  the  possibility  of  generating  a 
different  interpretation  of  the  scene  than  is  obtainable  by 
a  single  source  (Sohlberg  et  al .  ,  1994). 

The  type  of  data  fusion  being  performed  is  categorized 
based  on  at  which  step  during  processing  the  fusion  occurs. 
Pixel  level  fusion  occurs  when  the  fusion  is  performed 
between  the  base  images.  Feature  level  fusion  occurs  when 
features  are  extracted  from  the  images  before  fusion 
occurs,  e.g.,  using  segmentation  procedures.  Decision  level 
fusion  is  a  method  that  uses  information  extracted  from 
each  image  along  with  decision  rules  to  resolve  differences 
between  the  features  (Pohl  &  Genderen,  2010) . 

In  1990,  SPOT  panchromatic  image  data  were  fused  with 
SPOT  multispectral  data  for  visual  interpretation  using  an 
intensity-hue-saturation  (IHS)  transform  (Haydn  et  al., 
1982;  Carper  et  al .  ,  1990)  .  The  resulting  image  had  higher 
resolution  but  correlation  analysis  indicated  that  care 
must  be  used  in  the  analysis  of  the  resulting  spectral 
characteristics.  Pixel  level  fusion  is  not  limited  to 
fusion  of  remotely  sensed  images.  In  1989  radar  imagery  was 
combined  with  geophysical  data  into  an  IHS  image  that 
summarized  the  information  contained  in  both  (Harris  and 
Murray,  1999) .  Feature  level  fusion  has  been  shown  to  be  of 


use  in  creating  land  cover  maps  for  nuclear  incident 
response  (Sah  et  al .  ,  2012)  .  In  this  case  classification 

maps  were  created  from  multiple  sensors  and  then  merged  to 
create  a  final  classification  map  of  increased  accuracy.  In 
1995  decision  level  fusion  was  performed  using  neural 
networks  to  perform  unsupervised  classification  with  good 
results  (Baraldi  and  Parmiggiani,  1995).  Advantages  of 
this  approach  include  not  needing  a  priori  knowledge  of  the 
scene  and  fast  computation  times.  Because  no  a  priori 
knowledge  of  the  scene  is  used,  however,  there  is  no 
"correct"  classification  using  this  method  since 
classification  is  unsupervised  (Baraldi  and  Parmiggiani, 
1995)  . 

Data  fusion  between  different  sensors  often  requires 
working  with  different  spatial,  spectral,  or  temporal 
characteristics.  It  has  been  shown  that  the  use  of  data  of 
different  spatial  resolutions  can  be  beneficial  when 
applied  to  forest  mapping  (Kempeneers  et  al . ,  2011)  .  The 

effects  of  the  differing  acquisition  dates  of  the  datasets 
were  not  however  considered  for  the  purposes  of  this 
research . 
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Ill .  METHODS 


The  following  sections  outline  the  work  that  was 
performed  in  order  to  construct  the  fused  dataset  and  also 
the  classification  and  post-processing  evaluation  steps 
that  were  taken  to  validate  the  work. 


A.  FLAASH  ATMOSPHERIC  CORRECTION 


Atmospheric  correction  algorithms  are  used  on  spectral 
datasets  to  correct  or  compensate  for  the  effects  of 
atmospheric  propagation  in  remotely  sensed  data.  Fast  Line- 
of-sight  Atmospheric  Analysis  of  Spectral  Hypercubes 
(FLAASH) is  an  atmospheric  correction  algorithm  based  on  the 
MODerate  resolution  atmospheric  TRANsmission  (M0DTRAN4) 
atmospheric  transfer  model  developed  by  the  Air  Force 
Phillips  Laboratory,  Hanscom  AFB  and  Spectral  Sciences,  Inc 
(SSI)  (Cooley  et  al . ,  2002). 

The  goals  of  the  FLAASH  algorithm  are  to  provide 
accurate  atmospheric  properties  in  order  to  calculate  a 
correction  matrix  to  convert  the  radiance  value  recorded  at 
the  detector  to  surface  reflectance  (Cooley  et  al . ,  2002). 
In  order  to  do  this  the  FLAASH  uses  the  equation  below  to 
calculate  the  pixel  reflectance. 


L  = 


-d£_+_?£_+r 

1-ftS  i -p,s  " 


Equation  3 . 1 


where  L  is  the  radiance  at  a  sensor  pixel,  p  is  the  pixel 
surface  reflectance,  pe  is  the  surface  reflectance  averaged 
over  a  pixel  and  the  surrounding  region,  S  is  the  spherical 
albedo  of  the  atmosphere,  Z*  is  the  radiance  backscattered 
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by  the  atmosphere,  and  A  and  B  are  coefficients  determined 
by  the  atmospheric  and  geometric  conditions  of  the 
collection  and  are  calculated  using  the  MODTRAN  atmospheric 
model.  The  second  term  in  the  equation  makes  FLAASH  unique 
from  other  correction  algorithms  in  that  it  accounts  for 
adjacency  effects  (Matthew  et  al . ,  2002). 

The  WorldView-2  data  were  corrected  using  the  FLAASH 
algorithm.  Use  of  the  FLAASH  algorithm  requires  knowledge 
of  the  geolocation  of  the  image,  acquisition  date/time, 
pixel  size,  sensor  altitude,  ground  elevation,  and  sensor 
characteristics.  FLAASH  was  chosen  because  it  has  been 
shown  to  produce  comparable  results  to  competing 
atmospheric  correction  algorithms  while  accounting  for 
adjacency  effects  and  non-nadir  viewing  sensor  angles 
(Kruse,  F.  A.,  2004). 

B.  REGISTRATION 

The  superposition  process  requires  that  the  co¬ 
registration  between  two  images  have  very  little  error  for 
the  data  fusion  to  be  meaningful.  To  make  this  possible, 
relatively  small  chips  were  taken  from  the  original 
datasets.  The  WV-2  dataset  is  8,194  samples  and  7,135  lines 
at  2.25m  resolution.  A  chip  of  3,120  samples  and  2,363 
lines  was  taken  from  the  downtown  Monterey  area.  This  area 
was  selected  because  it  contains  numerous  distinct  features 
to  use  as  tie  points  for  the  registration  process  and  also 
contained  numerous  ground  cover  classes  for  the 
classification  step. 
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The  UAVSAR  dataset  are  3,300  samples  by  14,664  lines. 
These  data  were  chipped  down  to  2,400  samples  by  2,265 
lines  at  7  by  5  meter  resolution  over  an  area  containing 
the  entirety  of  the  WV-2  data  chip. 

Manual  selection  of  ground  control  points  (GCPs)  was 
performed  using  geographic  features,  road  intersections, 
and  building  corners.  In  total  74  GCPs  were  selected.  Using 
the  WV-2  data  as  the  base  image  and  warping  the  UAVSAR  data 
to  this  dataset,  the  two  images  were  co-registered  using 
the  triangulation  method  with  nearest  neighbor  resampling. 
After  warping  the  UAVSAR  data,  a  final  data  chip  was 
selected  to  correspond  in  both  size  and  space  with  the  WV-2 
resulting  in  a  3120  samples  by  2363  lines  at  2.25  meter 
resolution  dataset.  The  error  for  first  order  polynomial 
registration  using  the  selected  GCPs  (a  worst  case  estimate 
for  triangulation  error)  was  5.34  pixels.  The  final  data 
sets  are  shown  in  Figure  7 . 


Figure  7.  Final  registered  datasets.  Left:  WV-2,  Right: 

UAVSAR 
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C .  WV-2  CLASSIFICATION 

Classification  was  done  on  the  WV-2  dataset  using  the 
minimum  distance,  maximum  likelihood,  and  Mahalanobis 
distance  decision  rules.  Eight  regions  of  interest  (ROI) 
were  selected  using  a  true  color  composite  and  the  ground 
truth  was  verified  in  person  Figure  8  shows  a  true  color 
image  of  the  scene  with  the  ROIs  overlaid.  Each  site  was 

visited  individually  to  verify  via  inspection  that  the  ROIs 
represent  their  class  names.  The  red  ROI  is  a  selection  of 
urban  areas  between  roads.  This  ROI  has  a  lot  of  spectral 
variability  as  expected.  The  blue  ROI  is  an  arbitrary 
selection  of  the  ocean  in  the  image.  This  region  has 

relatively  low  spectral  variability.  The  green  region  is  a 
selection  of  "dense"  trees.  In  this  case  dense  means  an 
area  of  trees  that  is  dense  enough  to  prevent  visibility  of 
the  ground  from  above  at  2.25  meter  resolution.  The  yellow 
region  is  a  selection  from  a  couple  of  the  golf  courses  in 

the  area.  In  an  effort  to  limit  the  spectral  variability  of 

this  region,  only  fairway  grass  was  selected.  The  cyan 
region  is  a  selection  of  some  of  the  baseball  outfields  in 
the  image.  The  magenta  region  is  a  selection  of  the  sandy 
beaches.  These  beaches  can  have  low  to  high  spectral 

variability  depending  on  the  tides;  the  separation  in 
acquisition  dates  between  data  sets  may  affect  the  results 
more  significantly  in  this  region.  The  maroon  region  is  a 

selection  of  the  local  highways.  At  this  resolution 
vehicles  are  resolved  and  do  increase  the  spectral 

variability  for  this  region.  The  sea  green  region  is  a 

selection  of  "less  dense"  trees.  In  this  region  the  ground 
is  clearly  visible  in  between  the  trees. 
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Figure  8  is  an  image  of  the  Monterey  data  set  with  the 
selected  ROIs  overlaid.  Figure  9  illustrates  the  high 
spectral  variability  of  the  urban  region.  On  the  left  is  a 
plot  of  the  min/mean/max  spectra  from  the  urban  region.  The 
black  curve  is  the  mean  spectrum  from  the  ROI .  The  green 
curves  are  spectra  that  are  one  standard  deviation  from  the 
mean.  The  red  curves  are  the  upper  and  lower  envelope 
spectra  from  the  urban  region.  The  plot  on  the  right  is 
randomly  selected  spectra  from  the  urban  region. 


Figure  8 .  WV-2  true  color  image  of  the  Monterey  subset  with 

ROIs  overlaid 
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Figure  9.  Urban  ROI  mean/min/max  spectra  (left)  and  sample 

pixel  spectra  (right) 

Classification  results  for  all  classification 
approaches  applied  were  validated  using  the  confusion 
matrix  method  with  ground  truth  provided  by  ROI  selections 
for  the  same  classes  but  in  different  locations  in  the 
scene  as  shown  in  Figure  10.  The  ground  truth  was  verified 
locally . 
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Figure  10.  WV-2  data  with  reference  ROIs  overlaid 

D.  PAULI  DECOMPOSITION  COMPUTATION  AND  FUSION 

The  Pauli  decomposition  for  the  UAVSAR  data  was 
computed  using  the  the  formulas  described  in  Chapter  II  of 
this  work.  The  RGB  image  of  the  Pauli  decomposition,  where 
a  is  blue,  (3  is  red,  and  y  is  green,  is  shown  in  Figure 
11 . 
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Figure  11.  Pauli  decomposition  image 


The  image  shows  that  the  SAR  scene  contains  mostly 
volumetric  scattering  (green) .  There  are  almost  no  areas  of 
well-defined  single  (blue)  or  double  bounce  (red) 
scattering.  The  majority  of  areas  that  are  not  volumetric 
scattering  are  a  combination  of  single  and  double  bounce 
scattering,  which  results  in  the  pink  regions.  The 
dependence  of  SAR  data  on  the  angle  between  the  flight  line 
and  target  geometry  is  apparent  in  this  image.  Urban 
sections  with  buildings  and  streets  aligned  along  the  upper 
left  to  lower  right  diagonal  of  the  image  resulted  in  high 
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single  and  double  bounce  values.  Those  neighborhoods 
aligned  in  other  directions  returned  very  low  single  and 
double  bounce  values. 

The  multispectral  classification  methods  were  run  on 
the  three  band  Pauli  decomposition.  The  overall  accuracy  of 
this  classification  was  very  low  and  not  very  meaningful.  A 
short  analysis  of  one  of  the  three  classification  methods 
can  be  found  in  the  results  section  while  the 
classification  images  and  confusion  matrices  for  the  other 
two  methods  can  be  found  in  Appendix  A.  The  three  band 
image  that  resulted  was  concatenated  with  the  eight  band 
WV-2  data  which  resulted  in  an  image  with  11  dimensions  per 
pixel.  The  fused  11  dimension  dataset  was  classified  using 
the  same  three  algorithms  and  reference  data  as  the  WV-2 
dataset.  The  resulting  classification  images  were  used  to 
compute  confusion  matrices  for  comparison  with  the  WV-2 
classifications. 
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IV.  RESULTS 


The  WV-2  classification  results  are  shown  in  Figures 
13-19.  Figure  12  is  the  classification  map  legend  for 
reference  to  be  used  with  all  classification  images. 

Figure  13  is  the  multispectral  imagery  (MSI) 
Mahalanobis  distance  classification  image.  On  inspection, 
the  Mahalanobis  distance  classification  image  looks 
positive.  The  water  line  at  the  beach  is  very  distinct.  The 
less  dense  trees  region  on  the  left  side  of  the  image  is 
fairly  well  separated  from  the  dense  trees.  Baseball 
fields,  and  golf  courses  are  generally  well  classified. 
However,  specific  problem  areas  exist  in  this  image.  The 
water  classification  shows  errors  of  commission  on  four 
football  fields  throughout  the  image.  The  urban  region  is 
also  heavily  under-classified.  Also,  many  roads  are 
incorrectly  classified  as  urban.  The  beach  region  does  not 
appear  to  have  many  errors  of  omission,  however,  there  are 
many  urban  regions  incorrectly  classified  as  beach. 


■  Urban  [Redj  8455  points 

■  Water  [Bluej  33939  points 
Dense  Trees  [Greenj  9982  points 
Golf  Course  [Yellowj  6672  points 
Baseball  Outfield  [Cyanj  2600  points 
iBeach  [Magentaj  2800  points 

■  Road ./Highway  [Maroon]  5020  points 
[Less  Dense  Trees  [Sea  Greenj  6324  points 

Figure  12.  Classification  image  ROI  key 
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Figure  13.  MSI  Mahalanobis  distance  classification 

Figure  14  is  the  MSI  maximum  likelihood  classification 
image.  Despite  having  higher  overall  accuracy  based  on  the 
ground  truth  ROIs,  the  maximum  likelihood  classification 
image  appears  to  have  many  more  misclassif  ications .  The 
urban  class  has  a  number  of  false  positives  throughout  the 
image.  Most  obvious  is  the  misclassif  ication  of  the  ocean 
near  the  beach  as  urban  and  the  horseshoe  shaped  body  of 
water  in  the  middle  of  the  image.  Once  again,  golf  courses, 
baseball  fields,  and  roads  are  classified  well.  The  less 
dense  trees  are  well  classified  near  the  training  data  but 
not  elsewhere  in  the  image. 
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Figure  14.  MSI  maximum  likelihood  classification 

Figure  15  is  the  MSI  minimum  distance  classification 
image.  It  is  notable  that  the  "rough"  around  the  fairways 
in  the  golf  courses  were  classified  as  baseball  field  which 
should  be  considered  a  correct  classification.  However, 
this  image  appears  to  have  the  most  errors  throughout  the 
image.  There  are  a  lot  of  areas  classified  as  water 
throughout  urban  areas,  and  in  one  case  a  track  and  field 
track  is  classified  as  water.  The  beach  region  is  also  over 
classified  in  the  urban  areas.  Roads  are  under-classified, 
often  as  urban. 
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Figure  15.  MSI  minimum  distance  classification 

The  confusion  matrices  from  the  three  classification 
methods  are  shown  in  the  tables  below  in  percent  accuracy 
form.  For  readability,  class  names  were  changed  to  numbers 
corresponding  to  Table  4 . 


Class  name 

Class  number 

Urban 

1 

Water 

2 

Dense  trees 

3 

Golf  course 

4 

Beach 

5 

Less  dense  trees 

6 

Roads 

7 

Baseball  field 

8 

Table  4.  Confusion  matrix  legend 
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Table  5  is  the  confusion  matrix  for  the  Mahalanobis 
distance  classification  results.  Mahalanobis  distance 
classification  had  the  median  accuracy  and  kappa 
coefficient  of  71.73%  and  .6559  respectively.  The  degree  of 
agreement  for  this  classification  is  "good, "  according  to 
Table  3.  Statistics  to  note  within  the  table  include:  100% 
producer  accuracy  for  the  beach  region  and  near  100% 
producer  and  user  accuracies  for  the  water  region.  The 
producer  accuracy  was  especially  low  for  the  urban  as 
expected  because  of  the  number  of  omissions  apparent  in  the 
image.  The  user  accuracy  was  low  for  the  road  region  (27%) 
due  to  a  high  percentage  of  the  urban  region  being 
classified  as  road. 


Reference  Class 


Predicted 

Class 

1 

2 

3 

4 

5 

6 

7 

8 

Total 

1 

41.77 

0.01 

0.97 

0 

0 

0 

52.14 

0.53 

11.04 

2 

1.67 

99.98 

0.58 

0 

0 

0 

1.23 

0.05 

33.19 

3 

5.48 

0 

68.85 

0 

0 

0.07 

0.18 

41.74 

14.24 

4 

8.4 

0 

0 

100 

0 

0 

1.54 

0 

8.21 

5 

0.17 

0 

0.03 

0 

46.54 

20.49 

0 

2.45 

4.86 

6 

2.32 

0 

1.01 

0 

53.46 

78.89 

0.13 

4.29 

9.69 

7 

29 

0.01 

0.07 

0 

0 

0.07 

41.92 

0.1 

7.86 

8 

11.19 

0.01 

28.49 

0 

0 

0.47 

2.86 

50.85 

10.91 

Total 

100 

100 

100 

100 

100 

100 

100 

100 

100 

Table  5.  MSI  Mahalanobis  distance  confusion  matrix 


41 


Class 

Prod.  acc. 

User  acc. 

1 

41.77 

74.41 

2 

99.98 

98.57 

3 

68.85 

65.32 

4 

100 

78.92 

5 

46.54 

68.55 

6 

78.89 

50.25 

7 

41.92 

27.04 

8 

50.85 

42.94 

Table  6.  MSI  Mahalanobis  distance  user/prod.  acc. 

Table  7  is  the  confusion  matrix  for  maximum 
likelihood.  Maximum  likelihood  had  the  highest  overall 
accuracy  and  kappa  coefficient  of  83.73%  and  .8014 
respectively  which  indicates  a  "very  good"  degree  of 
agreement.  The  water,  dense  trees,  and  beach  regions  had 
the  highest  producer  accuracies.  The  beach  and  water 
regions  had  the  highest  user  accuracies.  The  road  region 
once  again  had  low  user  accuracy  (38%)  once  again  mostly 
because  of  misclassif ication  between  road  and  urban.  The 
most  significant  change  in  the  maximum  likelihood  results 
is  the  increase  in  baseball  field  user  accuracy  from  43%  to 
85%.  The  number  of  commissions  in  baseball  field 
classification  is  likely  the  result  of  a  drop  in  overall 
classification  percentage  from  10.9%  to  4.62%. 
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Reference  Class 


Predicted  Class 

1 

2 

3 

4 

5 

6 

7 

8 

Total 

1 

62.3 

0.09 

0.86 

0.06 

0.79 

2.13 

18.85 

0.82 

13.62 

2 

0 

99.74 

0 

0 

0 

0 

0 

0 

32.64 

3 

2.71 

0 

94.07 

0 

0 

0.18 

0.09 

47.6 

17.64 

4 

0.36 

0 

0.28 

83.1 

0 

5.13 

0 

8.52 

7.16 

5 

0.11 

0 

0 

0 

99.21 

0 

0 

0 

6.45 

6 

0.02 

0 

0 

16.83 

0 

92.56 

0 

0 

6.92 

7 

34 

0.17 

0.36 

0 

0 

0 

81.07 

0.41 

10.94 

8 

0.49 

0 

4.43 

0 

0 

0 

0 

42.64 

4.62 

Total 

100 

100 

100 

100 

100 

100 

100 

100 

100 

Table  7.  MSI  maximum  likelihood  distance  confusion  matrix. 


Class 

Prod.  acc. 

User  acc. 

1 

62.3 

89.99 

2 

99.74 

100 

3 

94.07 

72.02 

4 

83.1 

83.08 

5 

99.21 

99.65 

6 

92.56 

82.53 

7 

81.07 

37.54 

8 

42.64 

84.99 

Table  8.  MSI  maximum  likelihood  user/prod.  acc. 

Table  9  is  the  confusion  matrix  for  the  minimum 
distance  classification  results.  The  minimum  distance 
classification  rule  had  the  lowest  overall  accuracy  and 
kappa  coefficient  of  66.93%  and  0.5971  respectively.  This 
Kappa  statistic  indicates  a  "good"  degree  of  agreement.  The 
urban  and  roads  classes  had  very  low  individual  class 
accuracies  which  contributed  heavily  to  lower  overall 
accuracy  for  minimum  distance.  The  urban  and  less  dense 
trees  regions  had  notably  low  producer  accuracies  of  27% 
and  34%  respectively.  This  is  due  to  the  high  number  of 
commission  errors  from  several  other  classes  in  these  two 
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areas.  The  user  accuracies  ranged  from  23%  (road  region) 
to  94%  (water  region)  .  The  low  user  accuracy  for  the  urban 
class  is  correlated  with  the  low  producer  accuracies  for 
regions  with  commission  errors  in  urban  regions. 


Reference  Class 


Predicted  Class 

1 

2 

3 

4 

5 

6 

7 

8 

Total 

1 

26.69 

0 

0.07 

0 

0.14 

0.11 

55.97 

0.05 

8.12 

2 

4.71 

100 

8.99 

0 

0 

0 

0.04 

0.46 

34.91 

3 

6.27 

0 

67.05 

0 

0 

0.07 

0.4 

59.13 

15.76 

4 

0.01 

0 

0 

62.22 

0 

39.68 

0 

0 

6.9 

5 

21.29 

0 

0 

0 

99.86 

0.07 

1.32 

0 

10.74 

6 

2.32 

0 

0.35 

37.78 

0 

60.07 

0.04 

6.27 

7.5 

7 

35.94 

0 

0 

0 

0 

0 

41.83 

0.02 

9.19 

8 

2.77 

0 

23.55 

0 

0 

0 

0.4 

34.07 

6.88 

Total 

100 

100 

100 

100 

100 

100 

100 

100 

100 

Table  9.  MSI  minimum  distance  confusion  matrix 


Class 

Prod.  acc. 

User  acc. 

1 

26.69 

64.71 

2 

100 

93.74 

3 

67.05 

57.45 

4 

62.22 

64.49 

5 

99.86 

60.31 

6 

60.07 

49.46 

7 

41.83 

23.05 

8 

34.07 

45.59 

Table  10.  MSI  minimum  distance  user/prod.  acc. 


Figure  16  is  the  maximum  likelihood  classification  for 
the  SAR  Pauli  Decomposition  image.  Inspection  of  this  image 
indicates  a  low  overall  accuracy.  There  are  almost  no  urban 
or  road  classifications  at  all.  Beach  is  incorrectly 
classified  throughout  the  image  as  roads  and  urban.  The 
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golf  courses  and  baseball  fields  are  classified  as  water. 
Also  a  large  portion  of  the  image  is  incorrectly  classified 
as  golf  courses. 

The  corresponding  confusion  matrix  is  displayed  below 
in  Table  11.  The  overall  accuracy  and  Kappa  coefficient 
were  48.94%  and  .3666  respectively.  The  only  class  with 
high  accuracy  is  the  water  class;  however,  even  this  number 
is  skewed  by  the  sampling  of  the  ground  truth  ROI  method. 
The  image  indicates  lower  accuracy  than  98.87%.  The  golf 
course,  baseball  field,  and  roads  regions  had  producer  and 
user  accuracies  below  6%.  This  indicates  that  almost  every 
pixel  classified  as  one  of  these  classes  was  incorrect. 
These  results  indicate  that  general  multispectral 
classification  techniques  do  not  work  well  for  this  site  on 
the  SAR  data  alone.  The  remaining  classification  images  and 
confusion  matrices  for  the  SAR  classifications  can  be  found 
in  Appendix  A. 
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Figure  16.  SAR  maximum  likelihood  classification  image. 


Reference  Class 


Predicted  Class 

1 

2 

3 

4 

5 

6 

7 

8 

Total 

1 

24.04 

0 

0.79 

0 

0 

0.4 

3.04 

0 

5.02 

2 

0.02 

98.87 

0.02 

65.02 

42.5 

50.7 

13.03 

0.24 

43.58 

3 

6.37 

0 

32.92 

0.28 

0 

0.76 

2.77 

5.04 

6.37 

4 

21.92 

0 

22.23 

5.17 

0 

3.43 

14.53 

36.08 

11.96 

5 

10.15 

1.13 

0.81 

27.46 

57.5 

41.92 

59 

22.37 

15.81 

6 

9.31 

0 

0.91 

0 

0 

0.36 

1.67 

0.77 

2.13 

7 

12.45 

0 

3.82 

0.44 

0 

0.98 

1.06 

0.34 

3.14 

8 

15.75 

0 

38.5 

1.62 

0 

1.45 

4.89 

35.16 

11.99 

Total 

100 

100 

100 

100 

100 

100 

100 

100 

100 

Table  11.  SAR  maximum  likelihood  confusion  matrix 
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Class 

Prod. 

acc. 

User 

acc. 

1 

24.04 

94.31 

2 

98.87 

74.23 

3 

32.92 

69.78 

4 

5.17 

3.1 

5 

57.5 

23.58 

6 

0.36 

1.05 

7 

1.06 

1.7 

8 

35.16 

27.01 

Table  12.  SAR  maximum  likelihood  user/prod.  acc. 

After  pixel  level  fusion  was  performed,  the  same 
classification  algorithms  were  run  on  the  combined  data 
set.  The  classification  images  that  resulted  are  shown  in 
figures  17-19. 

Figure  17  is  the  Mahalanobis  distance  classification 
image  of  the  combined  dataset.  This  image  has  an  overall 
accuracy  of  74.34%  and  Kappa  coefficient  of  .6884.  This  is 
an  increase  of  almost  3%  in  accuracy  and  a  "good"  degree  of 
agreement . 

Inspection  of  the  Mahalanobis  distance  classification 
image  for  the  combined  dataset  reveals  the  most  prominent 
difference  between  the  MSI+SAR  and  the  MSI  classification 
to  be  an  increased  number  of  road  classifications  in  urban 


areas 

with  roads  aligned 

in  the 

left 

to  right 

direction . 

Also, 

there  is  an  increase  in 

urban 

classifications  in 

urban 

regions  where  the 

roads 

are 

aligned 

along  the 

diagonal  from  the  upper 

left  to 

the 

lower  ri 

ght  of  the 

image.  This  is  due  to  the  contributions  made  by  the  SAR 

data  in  areas  with  higher  single  and  double  bounce 

intensities.  There  is  also  a  small  increase  in  urban 

classifications  in  some  of  the  dense  trees  regions. 
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Figure  17.  MSI+SAR  Mahalanobis  distance  classification 

image . 


Figure  18  is  the  maximum  likelihood  classification 
image  that  resulted  from  the  combined  MSI+SAR  dataset. 
There  was  a  3%  decrease  in  overall  accuracy  (down  to  80.49% 
from  83.73%)  and  the  Kappa  coefficient  dropped  to  .7627. 
Despite  the  decrease  in  the  Kappa  coefficient  the  degree  of 
agreement  remains  "very  good." 

Despite  this  reduction  in  accuracy,  inspection  of  the 
image  indicates  some  areas  of  improved  classification  in 
this  image.  The  heavy  over  classification  of  the  urban 
region  that  resulted  in  the  initial  maximum  likelihood 
image  is  reduced.  For  example,  the  region  of  the  ocean  that 
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has  been  classified  as  urban  is  approximately  half  the 
width  as  it  was.  Many  dense  trees  areas  that  were 
incorrectly  classified  as  urban  are  now,  at  least  in  part, 
correctly  classified.  Also,  areas  of  the  beach  that  were 
incorrectly  classified  as  urban  are  now  more  properly 
classified  as  a  mix  of  beach  and  trees  (both  sparse  and 
dense) .  Inland  bodies  of  water  are  still  not  correctly 
classified.  Also  there  is  an  increase  in  the  number  of 
dense  tree  classifications  along  the  water  line.  These 
classifications  are  incorrect.  The  impact  of  the  line  of 
flight  of  the  SAR  data  is  less  apparent  in  this  example 
than  in  the  Mahalanobis  distance  classification. 


Figure  18.  MSI+SAR  maximum  likelihood  classification 

image . 
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Figure  19  is  the  minimum  distance  classification  image 


from  the 
identical 
did  not 
enough  to 


combined  MSI+SAR  dataset.  These  results  are 
to  the  MSI  results.  The  addition  of  the  SAR  data 
increase  the  separation  of  the  class  clusters 
change  any  classifications. 


Figure  19.  MSI+SAR  minimum  distance  classification 

image . 


The  corresponding  confusion  matrices  from  the  fused 
dataset  classification  images  are  in  tables  13,15,  and  17. 
Table  13  is  the  Mahalanobis  distance  confusion  matrix  for 
the  combined  dataset.  Overall  accuracy  improved  by  less 
than  3%  and  the  kappa  coefficient  improved  by  .03.  Neither 
of  these  are  significant  improvements.  However,  certain 
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classes  did  show  significant  improvement  in  producer  or 
user  accuracy.  The  road  class  showed  an  improvement  of  30% 
in  producer  accuracy  and  almost  20%  in  user  accuracy.  This 
is  due  to  the  increase  in  road  classifications  in  urban 
areas  in  general.  Roads  were  not  over-classified  in  the 
area  of  the  reference  data  because  the  reference  data  is  in 
an  area  with  high  single  and  double  bounce  intensities. 
This  resulted  in  higher  urban  classification  accuracy.  The 
urban  class  had  a  significant  increase  in  user  accuracy  of 
17%. 


Reference  Class 


Predicted  Class 

1 

2 

3 

4 

5 

6 

7 

8 

Total 

1 

44.76 

0 

0.3 

0 

0 

0.22 

15.15 

0.02 

9.63 

2 

1.7 

99.99 

0.64 

0 

0 

0 

1.45 

0.05 

33.22 

3 

5.2 

0 

69.76 

0 

0 

0.07 

0.18 

41.28 

14.26 

4 

0.19 

0 

0.03 

46.66 

0 

20.46 

0 

2.47 

4.87 

5 

6.26 

0 

0 

0 

100 

0 

2.07 

0 

7.82 

6 

2.26 

0 

1.09 

53.34 

0 

78.71 

0.13 

4.26 

9.67 

7 

28.72 

0.01 

0.05 

0 

0 

0.07 

77.5 

0.02 

9.59 

8 

10.91 

0.01 

28.13 

0 

0 

0.47 

3.52 

51.89 

10.94 

Total 

100 

100 

100 

100 

100 

100 

100 

100 

100 

Table  13.  MSI+SAR  Mahalanobis  distance  confusion  matrix 


Table  14 . 


Class 

Prod.  acc. 

User  acc. 

1 

44.76 

91.45 

2 

99.99 

98.5 

3 

69.76 

66.06 

4 

46.66 

68.54 

5 

100 

82.91 

6 

78.71 

50.27 

7 

77.5 

40.93 

8 

51.89 

43.72 

MSI+SAR  Mahalanobis  distance  user/prod.  acc. 
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Table  15  is  the  maximum  likelihood  confusion  matrix 
for  the  combined  dataset.  The  overall  statistics  for 
maximum  likelihood  also  showed  a  marginal  regression  of  3% 
in  overall  accuracy  and  .04  in  the  kappa  statistic.  Despite 
this,  there  were  notable  increases  in  producer  accuracy  for 
the  golf  course  and  road  regions  of  15%  and  11% 
respectively.  The  producer  accuracy  for  the  baseball  field 
region  decreased  by  almost  60%,  and  the  user  accuracy  for 
the  golf  course  region  decreased  by  30%.  This  is  due 
largely  to  a  high  correlation  between  the  omission  error 
pixels  for  golf  course  and  the  commission  error  pixels  for 
baseball  field. 


Reference  Class 


Predicted  Class 

1 

2 

3 

4 

5 

6 

7 

8 

Total 

1 

56.47 

0.01 

0.59 

0.09 

0.1 

1.26 

6.08 

0.07 

11.6 

2 

0 

99.93 

0 

0 

0 

0 

0 

0 

32.7 

3 

3.88 

0 

90.85 

0 

0 

0.18 

0.26 

34.36 

16.22 

4 

0.69 

0 

0.76 

98.66 

0 

65.23 

0 

20.36 

13.2 

5 

0.58 

0 

0 

0 

99.9 

0 

1.19 

0 

6.65 

6 

0.06 

0 

0 

1.25 

0 

33.21 

0 

0 

2.15 

7 

37.45 

0.06 

0.61 

0 

0 

0.11 

92.43 

0.27 

12.19 

8 

0.87 

0 

7.19 

0 

0 

0 

0.04 

44.94 

5.29 

Total 

100 

100 

100 

100 

100 

100 

100 

100 

100 

Table  15.  MSI+SAR  maximum  likelihood  confusion  matrix 
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Class 

Prod.  acc. 

User  acc. 

1 

56.47 

95.79 

2 

99.93 

100 

3 

90.85 

75.63 

4 

98.66 

53.48 

5 

99.9 

97.38 

6 

33.21 

95.33 

7 

92.43 

38.43 

8 

44.94 

78.35 

Table  16.  MSI+SAR  maximum  likelihood  user/prod.  acc. 


Table  17  is  the  minimum  distance  confusion  matrix  for 
the  combined  dataset.  There  was  no  change  in  the  results 
for  the  minimum  distance  classification  between  the  WV-2 
dataset  and  the  combined  dataset. 


Reference  Class 


Predicted  Class 

1 

2 

3 

4 

5 

6 

7 

8 

Total 

1 

26.69 

0 

0.07 

0 

0.14 

0.11 

55.97 

0.05 

8.12 

2 

4.71 

100 

8.99 

0 

0 

0 

0.04 

0.46 

34.91 

3 

6.27 

0 

67.05 

0 

0 

0.07 

0.4 

59.13 

15.76 

4 

0.01 

0 

0 

62.22 

0 

39.68 

0 

0 

6.9 

5 

21.29 

0 

0 

0 

99.86 

0.07 

1.32 

0 

10.74 

6 

2.32 

0 

0.35 

37.78 

0 

60.07 

0.04 

6.27 

7.5 

7 

35.94 

0 

0 

0 

0 

0 

41.83 

0.02 

9.19 

8 

2.77 

0 

23.55 

0 

0 

0 

0.4 

34.07 

6.88 

Total 

100 

100 

100 

100 

100 

100 

100 

100 

100 

Table  17.  MSI+SAR  minimum  distance  confusion  matrix 
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Class 

Prod. 

acc. 

User 

acc. 

1 

26.69 

64.71 

2 

100 

93.74 

3 

67.05 

57.45 

4 

62.22 

64.49 

5 

99.86 

60.31 

6 

60.07 

49.46 

7 

41.83 

23.05 

8 

34.07 

45.59 

Table  18.  MSI+SAR  minimum  distance  user/prod.  acc. 

Table  19  is  a  summary  of  the  classification  results' 
overall  accuracies  and  Kappa  coefficients  before  and  after 
fusion.  The  overall  accuracies  and  Kappa  coefficients  did 
not  change  much  after  fusion,  however,  inspection  of  the 
classification  images  indicated  improvement  in  the 
classifications  not  shown  in  the  confusion  matrices.  These 
improvements  were  particularly  evident  for  the  urban 
regions  distinguished  in  the  Pauli  decomposition  image.  The 
negligible  and  sometimes  negative  change  in  accuracy 
despite  the  apparent  improvement  in  accuracy  is  likely  due 
to  the  limited  scope  of  the  method  of  ROI  selection  used  to 
quantify  the  error.  Using  ground  truth  ROIs  instead  of  a 
ground  truth  image  approximates  the  error  using  a  sample. 


MSI  Classification 

MSI+SAR  Classification 

Mahalanobis  distance 

Accuracy 

71.73% 

74.34% 

Kappa  coefficient 

0.6559 

0.6884 

Maximum  likelihood 

Accuracy 

83.73% 

80.49% 

Kappa  coefficient 

0.8014 

0.7627 

Minimum  distance 

Accuracy 

66.92% 

66.92% 

Kappa  coefficient 

0.5971 

0.5971 

Table  19.  Results  summary  table. 
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V.  CONCLUSIONS 


Data  from  the  UAVSAR  and  WorldView-2  sensors  of 
Monterey,  CA  were  successfully  fused.  VNIR  multispectral 
data  were  analyzed.  A  common  SAR  decomposition,  the  Pauli 
decomposition  was  performed  and  inspected.  The  SAR  Pauli 
decomposition  and  the  multispectral  reflectance  data  were 
fused  at  the  pixel  level,  then  analyzed  using  multispectral 
classification  techniques.  The  results  were  compared  to  the 
multispectral  classifications  using  the  SAR  decomposition 
results  for  a  basis  of  interpreting  the  changes.  The 
resulting  confusion  matrices  showed  little  to  no 
improvement  in  accuracy  based  on  the  samples  used  for 
reference  ROIs,  however,  visual  inspection  of  the 
classification  images  revealed  a  subjective  increase  in 
overall  accuracy  in  two  of  the  maps  (Mahalanobis  distance 
and  maximum  likelihood) .  The  minimum  distance  classifier 
experienced  essentially  no  change  in  results  from  the 
addition  of  the  SAR  information. 

The  most  noticeable  increases  in  classification 
accuracy  occurred  in  spatial  regions  where  the  land 
features  were  parallel  to  the  SAR  flight  line.  This 
dependence  on  orientation  makes  this  fusion  process  more 
ideal  for  datasets  with  more  consistent  features  throughout 
the  scene. 

Future  ways  to  expand  on  this  work  include  using 
higher  spatial  resolution  SAR  data  and  adjusting  the  weight 
of  the  SAR  data  in  the  fusion  process.  Increased  spatial 
resolution  for  the  SAR  data  will  provide  more  detail  for 
some  of  the  finer  regions  of  interest.  For  example,  the 
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golf  course  and  road  regions  are  very  narrow  by  design.  In 
some  places  these  regions  were  as  narrow  as  five  to  ten 
meters.  In  the  original  SAR  data  (before  resampling  during 
registration)  these  regions  would  have  been  one  to  two 
pixels  wide.  Because  SAR  data  describes  texture  and 
geometry  of  surface  features  regions  of  interest  should 
ideally  be  larger.  Another  reason  that  the  fusion  of  the 
SAR  data  may  have  had  insignificant  effects  on 
classification  may  have  been  the  weight  of  the  data  in  the 
classification  algorithm.  Because  all  algorithms  used  in 
this  work  weigh  each  dimension  of  the  vector  to  be 
classified  equally,  the  SAR  data  contributed  less  than  half 
of  the  weight  of  the  WV-2  data  to  the  final  vector  (three 
dimensions  versus  eight  respectively) .  Giving  the  SAR  data 
more  weight  in  the  classification  process  would  increase 
the  impact  in  the  results  of  the  additional  data. 
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APPENDIX:  ADDITIONAL  RESULTS  FIGURES  AND  TABLES 


Figure  20.  SAR  Mahalanobis  distance  classification 


Reference  Class 


Predicted  Class 

1 

2 

3 

4 

5 

6 

7 

8 

Total 

1 

36.55 

0 

1.22 

0.28 

1.37 

0 

2.64 

0.05 

7.6 

2 

4.58 

100 

0.74 

65.12 

30.36 

10.46 

39.76 

4.79 

43.39 

3 

16.04 

0 

46.12 

0.34 

0.94 

0 

4.58 

12.54 

10.86 

4 

9.84 

0 

18.3 

2.09 

1.77 

1.79 

4.71 

16.49 

6.54 

5 

6.04 

0 

4.59 

1.12 

0.69 

0 

4.58 

6 

2.72 

6 

4.56 

0 

1.44 

28.93 

62.81 

87.75 

34.79 

23.46 

16.65 

7 

14.78 

0 

5.85 

1.28 

0.72 

0 

6.91 

13.58 

5.44 

8 

7.6 

0 

21.74 

0.84 

1.34 

0 

2.03 

23.08 

6.8 

Total 

100 

100 

100 

100 

100 

100 

100 

100 

100 

Table  20.  SAR  Mahalanobis  distance  confusion  matrix 
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Table  2 1 . 


Class 

Prod. 

acc. 

User 

acc. 

1 

36.55 

94.63 

2 

100 

75.41 

3 

46.12 

57.38 

4 

2.09 

2.9 

5 

0.69 

1.56 

6 

87.75 

34.16 

7 

6.91 

6.44 

8 

23.08 

31.26 

SAR  Mahalanobis  distance  user/prod.  acc. 


Figure  21.  SAR  minimum  distance  classification  image 
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Reference  Class 


Predicted  Class 

1 

2 

3 

4 

5 

6 

7 

8 

Total 

1 

33.84 

0 

1.59 

0.25 

0.4 

0 

1.23 

0 

6.98 

2 

1.35 

89.92 

0.38 

87.84 

84.24 

64.11 

59.14 

13.34 

49.61 

3 

28.11 

0 

47.94 

0.9 

2.39 

0 

9.47 

18.01 

14.36 

4 

6.97 

0 

4.89 

2.43 

1.55 

1.89 

4.27 

7.38 

3.32 

5 

7.76 

0 

14.35 

0.62 

0.51 

0 

5.55 

14.04 

5.12 

6 

7.69 

10.08 

1.17 

6.42 

8.93 

34 

15.72 

14.02 

10.27 

7 

3.06 

0 

2.56 

0.31 

0.54 

0 

2.25 

4.65 

1.55 

8 

11.22 

0 

27.12 

1.22 

1.45 

0 

2.38 

28.55 

8.8 

Total 

100 

100 

100 

100 

100 

100 

100 

100 

100 

Table  22.  SAR  minimum  distance  confusion  matrix 


Class 

Prod. 

acc. 

User 

acc. 

1 

33.84 

95.43 

2 

89.92 

59.31 

3 

47.94 

45.1 

4 

2.43 

5.24 

5 

0.51 

0.61 

6 

34 

21.45 

7 

2.25 

7.36 

8 

28.55 

29.89 

Table  23. 


SAR  minimum  distance  user/prod,  acc . 
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